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Executive Summary 

The 21st Century Community Learning Centers (21st CCLC) grant program provides 
academic enrichment opportunities outside of the regular school day to help students meet state 
and local performance standards in core academic subjects. This report summarizes the results of 
the Center for Research in Educational Policy’s evaluation of the 2017-2018 Virginia 21st CCLC 
programs. The purpose was to determine whether the federally-funded 21st CCLC programs 
were meeting Virginia’s program objectives by: (1) improving student academic achievement in 
reading; (2) improving student academic achievement in mathematics; and (3) providing 
opportunities for parental education. While not one of the statewide objectives, an analysis of 
school-day attendance was also included. 

Data were analyzed from five main sources: 1) 21st CCLC and school day attendance 
data on all student participants with available SOL scores; 2) scores for reading and mathematics 
from the Standards of Learning (SOL) assessments, Virginia Alternate Assessment Program 
(VAAP), and Virginia Grade Level Alternative (VGLA) assessment, 3) the online Annual Local 
Evaluation Survey (ALERT); 4) the Student Perceptual Survey, and 5) the Teacher Annual 
Performance Report (APR) Survey. 

Conclusions 

The conclusions of the analyses are summarized below by evaluation question. 

1. What is the nature of the Virginia 21st CCLC grant program and level of participation by 
students? 

Both 21 CCLC students and their regular school-day teachers reported positive views of 
the program. Specifically, students attending 30 hours or more were asked to provide their 


perceptions of why they attended the 21st CCLC afterschool program, their general impression 
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of the 21st CCLC program, and to report any benefits they attribute to 21st CCLC program 
attendance. 

When asked why they attended the 21st CCLC afterschool programs, about half of the 
students indicated “the afterschool program helps me do better in school” (59%), “the activities 
are fun” (55%), and “I like going to the afterschool program” (54%). A sizable majority of 
students also indicated that during the afterschool program they “feel safe” during 21st CCLC 
programs (78%), “the staff encourages them to do their best” (74%), and “they trust the staff” 
(74%). Moreover, the majority of students felt the 21st CCLC afterschool program has helped 
them to “prepare for trade school or college” (73%), “prepare for a job or a career” (72%), “get 
better grades in school” (72%), and “be better at math” (71%). 

As for school-day teachers, those who participated in the online Teacher APR Survey (a 
reflection on how each CCLC student is doing) indicated that 21*t CCLC student behavior mostly 
Improved or Did Not Need to Improve in six key behavioral areas. Specifically, 21‘ CCLC 
student behavior Jmproved in “academic performance” (63%), “motivation to learn” (54%), 
“turning in his/her homework on time” (53%), and “getting along well with other students” 
(41%). Largely, school-day teachers indicated an overall improvement for student’s in 
homework completion and class participation (76%) and classroom behavior (69%). 

The level of participation was similar to prior years. For the 2017-2018 school year, there 
were 3,803 paid and volunteer staff members in 136 Virginia 21st CCLC centers. School-day 
teachers, nonteaching school staff, and center administrators or coordinators comprised the 
majority of the paid staff, while the volunteer staff included college and high school students, 


community members, retired teachers, and parents. 


Virginia 21 CCLC 2017-2018 Evaluation 4 


There were 9,986 students attending centers regularly (i.e., for a minimum of 30 days) 
during 2017-2018. Students served were in pre-kindergarten through grade twelve, with the 
majority in grades three through eight. The majority were African American (44%), White 
(34%), or Hispanic (14%), and over three-fourths of all students were at an economic 
disadvantage (78%). A small portion had limited English proficiency (LEP) (11%) and special 
needs or disabilities (15%). When compared to the state, the 21st CCLC program had a much 
higher number of students at an economic disadvantage and they served a larger percentage of 
African Americans (44% ED and 22% AA respectively). 

2. To what degree did centers meet Virginia’s objectives for the program? 
Objective 1: Improve Student Academic Achievement in Reading 

For students in grades four through eight, the proficiency level and standardized SOL 
scaled score analyses showed that there was no statistically significant impact of 21‘ CCLC 
participation (“Yes” or “No’”’) on statewide reading assessments for the overall samples (i.e., all 
students combined). Additionally, the effect size for the proficiency level analysis (Cox Index 
effect size (CIES) = -0.01) and for the standardized SOL scaled score analysis (g = 0.00), which 
quantifies the magnitude of the difference, would not be considered substantively important (1.e., 
educationally meaningful) based on What Works Clearinghouse (WWC) guidelines (> +/- 0.25) 
(What Works Clearinghouse, 2017). While there were no statistically significant or 
substantively important differences in reading proficiency level for the Limited English 
Proficient (LEP) or Economically Disadvantaged (ED) subgroups based on 218 CCLC 
participation, students in 218‘ CCLC receiving special education services (SPED) statistically 
significantly outperformed SPED controls in reading proficiency level outcomes, although the 


effect was not substantively important. There were no statistically significant or substantively 
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important differences in reading standardized SOL scaled scores for the ED subgroups based on 
21 CCLC participation. While not tested for statistical significance, for students in grade three 
who did not have prior-year test scores available, 21° CCLC participants in 2017-2018 were 
outperformed in reading proficiency by (a) non-participants for all students combined and on 13 
out of 15 available subgroupings and by (b) Virginia for all students combined and 14 of 15 
available subgroupings. In terms of SOL reading scaled scores, third-grade 21*t CCLC 
participants in 2017-2018 were outperformed by (a) non-participants overall and in 12 out of 15 
subgroup comparisons and by (b) Virginia for all students combined and 14 of 15 available 
subgroupings. 
Objective 2: Improve Student Academic Achievement in Mathematics 

For students in grades four through twelve, the proficiency level and standardized SOL 
scaled score analyses showed no statistically significant impact of 21‘ CCLC participation 
(“Yes” or “No’’) on statewide mathematics assessments for the overall sample (i.e., all students 
combined). Additionally, the effect sizes for the proficiency level analysis (Cox Index effect size 
(CIES) = 0.04) and for the standardized SOL scaled score analysis (g = 0.03) would not be 
considered substantively important based on What Works Clearinghouse (WWC) guidelines 
(= +/- 0.25). There were no statistically significant or substantively important differences in 
mathematics proficiency level for the LEP or ED subgroups based on 21*' CCLC participation. 
However, SPED students in 21° CCLC statistically significantly outperformed SPED control 
students in mathematics proficiency level outcomes, although the effect was not substantively 
important. In addition, 218‘ CCLC ED students statistically significantly outperformed ED 
control students on mathematics standardized SOL scaled scores, while non-ED control 


students statistically significantly outperformed non-ED 21*t CCLC students on mathematics 


Virginia 21° CCLC 2017-2018 Evaluation 6 


standardized SOL scaled scores, although the effects were not substantively important. Although 
not tested for statistical significance, for students in grade three who did not have prior-year test 
scores available, 21°‘ CCLC participants in 2017-2018 were outperformed in proficiency by non- 
participants on 5 out of 15 subgroupings and Virginia on all students combined, as well as all but 
three subgroupings. In terms of SOL scaled scores, third-grade 21*' CCLC participants in 2017- 
2018 were outperformed by non-participants overall and in 7 out of 15 subgroup comparisons in 
mathematics and Virginia on all students combined, as well as all but two subgroupings. 

Improve Student School-Day Attendance. For students in grades four through twelve 
who participated for at least 30 days, the school-day attendance analyses showed a statistically 
significant positive impact of 21‘ CCLC participation (“Yes” or “No”) on school-day attendance 
for the overall (i.e., combined) sample. However, the effect size (g = 0.06) would not be 
considered substantively important. In addition, there were statistically significant, but not 
substantively important positive differences in school-day attendance favoring 218‘ CCLC 
participants over control students for the not LEP, SPED, not SPED, and ED subgroups. 

Objective 3: Provide Opportunities for Parent Education 

As a whole, 4,800 parents and adult family members were served during the 2017-2018 
school year, including 537 parents and adult family members participating over the summer. 
Activities were specifically chosen by grantees to provide “opportunities for parent/child 
interaction in academic activities” (84%), “parenting skills training” (51%), “health and wellness 
training” (41%), “career development information/training” (25%), “computer skills instruction” 
(22%), “General Educational Development (GED) certificate program” (13%), and “Other” 
subobjectives and activities (12%). Grantees reported meeting their selected subobjectives as 


follows: “opportunities for parent/child interaction in academic activities” (97%), “parenting 
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skills training” (97%), “career development information/training” (96%), “health and wellness 
training” (96%), “Other” subobjectives and activities (92%), “computer skills instruction” 
(92%), and “General Educational Development (GED) certificate program” (53%). 
3. In what ways do attendance at a 21st CCLC, type and time allocated to activities, and hours 
of operation predict academic achievement? 

Statistical analyses of associations between various categories of center-level data and 
reading and mathematics outcomes of 21‘ CCLC students in grades four through eight (grades 
four through twelve for mathematics) with a minimum of 30 days of attendance and two years of 
assessment data available provide information that may be useful to program leaders and are 
summarized below. 

Center-level results from analysis of reading outcomes. The total number of staff at 
the center had a small, but positive statistically significant impact on both reading proficiency 
level and standardized SOL reading scaled scores in 2017-2018. The percentage of math related 
activities and the percentage of paid certified school-day teachers had a very small, positive 
impact on both reading proficiency level and standardized SOL reading scaled scores. 
Meanwhile, total hours of activities, the total number of activities, and the percent of activities 
that were academic, all had very small, non-statistically significant impacts on both reading 
proficiency level and standardized SOL reading scaled scores in 2017-2018. No center level 
variables had a substantively impact on either reading proficiency level or standardized SOL 
reading scores. 

Center-level results for mathematics. No center level variables had a statistically 
significant or substantively important impact on either mathematics proficiency level or 


standardized SOL mathematics scores in 2017-2018. However, the percent of activities that were 
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math related, as well as the total number of staff had a very small, positive, but non-statistically 
significant impact on both mathematics proficiency level and standardized SOL mathematics 
scores. Meanwhile, the total number of hours of activities, the percent of activities that were 
English related, the percent of academic activities, and the percent of enrichment activities all 
had very small, non-statistically significant negative impacts on both mathematic proficiency 
level and standardized SOL mathematics scaled scores in 2017-2018. 

4. What “promising practices” were identified by centers regarding the achievement of 

required objectives? 

Grantees were asked to elaborate upon activities or promising practices that appeared 
most effective in efforts to meet their subobjectives in each of six areas, as well as their “other” 
objectives, and to make recommendations for improvements to the program. Major themes 
appearing in grantees’ responses included: (a) Providing academic support in core areas, 
especially in math and reading/English; (b) employing qualified core subject teachers in the 
afterschool programs and ensuring effective communication and coordination between 
afterschool and day school staff; (c) providing a broad variety of student enrichment activities; 
(d) providing parent education to enhance parents’ ability to assist their children, as well as their 
employability and their life skills; (e) relationship-building with families of students through 
consistent communication and engagement; (f) instructional methods, activities, and techniques 
for effective character education to improve school climate and student academic, social, and 
emotional skills; and (g) maintaining strong relationships and communication between program 


staff and community partners. 
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Introduction 

This report presents the evaluation by the Center for Research in Educational Policy 
(CREP) of the 2017-2018 Virginia 21*t Century Learning Centers’ (21*‘ CCLC). The mixed- 
methods evaluation utilized perceptual data from study participants as well as program and 
school-day attendance data, and proficiency level and scaled scores from reading and math 
statewide assessments. 

CREP is a State of Tennessee Center of Excellence, located at The University of 
Memphis, whose mission is to implement a research agenda associated with educational policies 
and practices in preK-16 schools and to provide a knowledge base for use by educational 
practitioners and policymakers. Since 1989, the Center has served as a mechanism for mobilizing 
community and university resources to address educational problems and to meet the 
University’s commitment to primary, secondary, and higher education institutions. Functioning 
as part of the College of Education, the Center seeks to accomplish its mission through a series 
of investigations conducted by Center faculty, staff, and associates, College and University 
faculty, and graduate students. 

Background and Program Description 

The 21st Century Community Learning Centers (CCLC) grant program was established 
by Congress as Title X, Part I, of the Elementary and Secondary Education Act (ESEA). It was 
reauthorized by Congress under the Every Student Succeeds Act of 2015 (ESSA). The purposes 
of the 21st CCLC program are as follows: 

e To provide academic enrichment opportunities outside of the regular school day to help 
students, particularly students who attend high-poverty and low-performing schools, meet 


state and local performance standards in core academic subjects. 
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e To offer students a broad array of services, programs, and activities to complement 
academics such as drug and violence prevention; counseling programs; art, music and 
recreation programs; technology education; and character education. 
e To offer families of students served by community learning centers opportunities for literacy 
and related educational development. 
21°* Century Community Learning Centers in Virginia 

Every year applicants apply for competitive 21st CCLC grant funds from the Virginia 
Department of Education (VDOE). For the 2017-2018 evaluation, which includes centers within 
a three-year grant cycle (Cohorts 14 — 16), there were 118 grantees that operated programs in 
136 schools and/or centers. The grantees provided academic and enrichment programs to 
students before and/or after school, and some offered programs during the summer as well. The 
program also supported grantee collaboration with parents and community partners. 
Evaluation Objectives and Questions 

States, as the recipients of 21‘ CCLC funds, are responsible for providing comprehensive 
evaluations of their programs. CREP was contracted by the Virginia Department of Education to 
conduct a statewide evaluation of the 21** CCLC program and to assess the extent to which local 
grantees met the following defined programmatic objectives. 

Objective 1: Improve student academic achievement in reading. 

Objective 2: Improve student academic achievement in mathematics. 

Objective 3: Provide opportunities for parental education. 

To address the 21‘ CCLC objectives, CREP’s evaluation is structured around the following 


questions: 
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1. What is the nature of the Virginia 21*' CCLC grant program and level of participation by 
students? 
2. To what degree did centers meet Virginia’s objectives for the program? 
3. In what ways do attendance at a 21 CCLC program, the types of and time allocated to 
activities, and hours of operation predict academic achievement? 
4. What “promising practices” regarding the achievement of required objectives were 
identified? 
Method 
Participants 
The 2017-2018 population consisted of 21‘ CCLC grantees in Virginia, school-day 
teachers at participating schools, student participants in the center programs, and the parents of 
student participants. The study population, along with others associated with the program, are 
discussed in detail below in the report section entitled Center and Participant Characteristics. 
Instrumentation 
Data were analyzed from five main sources: (a) 21st CCLC and school-day attendance 
data on all student participants with available SOL scores; (b) scores for reading and 
mathematics from the Standards of Learning (SOL) assessments, Virginia Alternate Assessment 
Program (VAAP), and Virginia Grade Level Alternative (VGLA) assessment, (c) the online 
Annual Local Evaluation Survey (ALERT); (d) the Student Perceptual Survey, and (e) the 
Teacher Annual Performance Report (APR) Survey. These sources are summarized by 
evaluation question in Table 1 below. It should be noted that 2016-2017 was the last year the 


VGLA was administered. 
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Table 1. Summary of Instruments and Data Sources by Evaluation Question 
Evaluation Question Data Sources 


What is the nature of the 21st CCLC programs and level of carte 
ant Student Perceptual Survey 
participation by students? 


Teacher APR Survey 


ALERT 
To what degree did centers meet Virginia’s objectives for Virginia SOL test scores in reading and 
the program? mathematics 
21st CCLC and school day attendance data 
. ALERT 
In what ways do attendance at a 21st CCLC, type and time is : . 
soy : ; Virginia SOL test scores in reading and 
allocated to activities, and hours of operation predict athemalioe 


; ; s 
academic echiyyereat) 21st CCLC and school day attendance data 


What “promising practices” regarding the achievement of 


required objectives were identified by centers? ee 


Surveys 

The ALERT, Student Perceptual Survey, and Teacher APR Survey were administered 
through CREP’s online Survey Management System (SMS). Online tools and surveys were 
opened for one to three months (depending on the instrument) and grantees were asked for full 
participation in order to get a complete picture of the 21*‘ CCLC program throughout Virginia. 

ALERT. The ALERT is an extensive online tool a grantee is required to submit for each 
center after a full year of program implementation. Its purpose is to gather additional data 
regarding a center’s measurable objectives, activities, and outcomes. Each grantee was asked to 
fill out one ALERT for each site, and to include only data pertaining to students who attended 
programs for 30 days or more. As part of the self-reporting in ALERT, grantees also were asked 
to describe the “promising practices” that they had found most helpful in an effort to meet their 
subobjectives for each of seven specified areas, and to provide recommendations for 
improvements to the program. It is important to note that grantees determined and self-reported 


their individual levels of success in meeting objectives based on their own criteria. 
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Student Perceptual Survey. The Student Perceptual Survey was developed to give 
students the opportunity to anonymously provide their perceptions of the 21st CCLC program 
and a means to report benefits they attribute to their program attendance. It is administered in 
both English and Spanish in order to capture a wider student population. 

Teacher APR Survey. The Teacher APR Survey was designed to collect information 
from the regular school-day teacher about changes in class attendance, academic performance, 
behavior, and homework completion for each 21st CCLC student. 

State Assessment and Attendance Data 

VDOE also provides CREP data to analyze how the program impacts student 
achievement in reading and math, as well as school-day attendance. 

Data were examined using Hierarchical Linear Models (HLM) and Hierarchical 
Generalized Linear Models (HGLM) for students with two years of test data available in grades 
four through eight for reading, and grades four through twelve for mathematics. Analyses of the 
impacts of center-level factors (e.g., the number of hours of center activities) on student 
achievement only included students who participated in 21‘' CCLC for 30 or more days (i.e., no 
control students were included). Additional HLM and HGLM models were examined by 
comparing matched pairs of students in the treatment group who attended 21*t CCLC programs 
for 30 or more days (i.e., students meeting the definition of “regular attendee”) and students in a 
control group who were eligible to attend 21‘ CCLC programs but had zero days of attendance. 
A similar matched pairs analysis was performed using HLM to determine the impact of 21* 
CCLC attendance on school-day attendance. 

Two sets of analyses (eight analyses total): Four for proficiency-level, and four for 


standardized SOL scaled scores were conducted separately by subject area (reading and 


Virginia 21° CCLC 2017-2018 Evaluation 14 


mathematics): Proficiency-level: (1) reading and (2) mathematics for 21*' CCLC participants vs. 
controls, (3) reading and (4) mathematics for 21*' CCLC participants only (center impacts); SOL 
Scaled Scores: (5) reading and (6) mathematics for 21*' CCLC participants vs. controls, (7) 
reading and (8) mathematics for 21‘' CCLC participants only (center impacts). The first set of 
analyses assessed proficiency-level performance in 2017-2018 based on all available test data 
(i.e., SOL, VAAP, and VGLA) using HGLM. For these analyses, the proficiency level on the 
SOL, VAAP, or VGLA test for the 2016-2017 and 2017-2018 school years was treated as either 
“pass” (based on scoring “Proficient,” “Advanced Proficient,” or “Advanced/College Path”’), or 
“fail” (based on scoring “Basic” or “Below Basic”). This method permitted the inclusion of all 
students, regardless of the type of assessment taken to participate in Virginia’s statewide testing 
program (i.e., traditional or alternative), as proficiency level is a common measure across each of 
the different test types, grade levels, and years. Center-level variables (e.g., total number of 
center activities) were included in specified analyses to examine the impacts of these variables 
on student proficiency. By including all students in the analyses, this method offers the most 
appropriate tool to analyze outcomes for specific student subgroups for the data available. 

The first proficiency analyses investigated the relationship of 21** CCLC participation on 
student achievement. 21‘ CCLC students who participated for at least 30 days and matched 
control students (who were eligible but did not participate in 21‘' CCLC) were included 
(combined n = 11,396 for reading, combined n = 12,022 for mathematics). Additionally, the 
effects of 21‘ CCLC participation on three subgroups, based on special education status (SPED), 
Limited English Proficiency (LEP) status, and Economically Disadvantaged (ED) status, were 


examined. The second proficiency analyses investigated the relationship of center-level 
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characteristics on student achievement for only 21‘' CCLC students who participated for at least 
30 days (n = 5,698 for reading, n = 6,011 for mathematics). 

While these analyses were designed to capture broad impacts on student proficiency 
associated with participation in the 21‘‘ CCLC programs, these analyses were not designed to 
measure incremental differences in student achievement or differences between treatment and 
control students that may occur within proficiency levels. For example, students who initially 
scored at the low end of proficiency, but moved to the high end of proficiency, would have 
demonstrated no measurable change in the proficiency analyses because their overall proficiency 
level (i.e., Proficient or Not Proficient) had not changed, even though their academic 
achievement may have increased from one year to the next. 

Therefore, the next two sets of analyses focused on the standardized scaled scores of 
students who took the SOL assessments in both 2016-2017 and 2017-2018, using HLM. These 
SOL analyses were intended to be more sensitive to these types of changes that occur across the 
scaled score range, regardless of students’ proficiency levels. The standardized SOL scaled 
score analyses included the same student-level and center-level variables used in the proficiency 
level analyses, and in terms of student subgroups, looked at the effects of 218 CCLC 
participation by ED status only (vs. ED, SPED, and LEP), for reasons explained in the following 
paragraph. 

The first set of SOL analyses investigated the relationship between 21° CCLC 
participation and student achievement for matched 21‘ CCLC and control students (combined 
n= 11,166 for reading, combined n = 11,958 for mathematics). Additionally, the effect of 21* 
CCLC participation by economically disadvantaged status was examined. The second set of 


SOL analyses investigated the relationship of center-level characteristics on student achievement 
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for 21‘ CCLC students who participated for at least 30 days (n = 5,583 for reading, n = 5,979 for 
mathematics). It is important to note that while the scaled score analyses were potentially more 
sensitive to changes attributable to program participation, they also had limitations. In particular, 
because students who participated in alternative assessments were not included (as scaled scores 
on SOL and alternative tests are not comparable), this type of analysis should not be used to 
evaluate the impact of participation in the 21‘ CCLC program on students receiving special 
education services and students with limited English proficiency, as the SOL assessment 
outcomes for these two subgroups would not be representative of the total population of SPED 
and LEP students, and subsequently were not included in the SOL analyses. 

Furthermore, as Virginia’s tests are not vertically scaled, meaning that scores from 
different tests, grade levels, and years are not directly comparable in terms of measuring the 
amount of learning, the test-level' outcome data were converted to standardized scores (i.e., Z- 
scores) prior to analysis. As a result, the data were placed onto a single, comparable scale while 
retaining the shape of the distribution of the original scores. The conversion also allowed 
different grade levels to be combined so that the effectiveness of centers could be evaluated 
based on all students served. While this transformation is the best available approach to measure 
achievement using scaled scores from multiple grades in Virginia at this time, the conversion has 
limitations as z-scores only provide a measure of achievement relative to Virginia’s average, and 
are not a measure of absolute growth or change from year to year. Thus, the full implications of 
this conversion applied to Virginia’s criterion-referenced tests are not clear. 

The attendance analyses investigated the impact of 21*' CCLC participation on students’ 
school-day attendance. For each student, the number of days present and the number of days 


' The test level is the achievement test level independent from grade level. Therefore, students’ scores were 
standardized based on the test level of the test they took, not the grade level in which they were enrolled. 
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absent had to total the number of days in session, otherwise the record was deleted. The number 
of days present was divided by the number of days in session to get a percent of days present for 
each student. This allows all students to be compared using the same scale, regardless of any 
variation in the number of days each school was in session. Matched 21 CCLC students who 
participated for at least 30 days and control students (who were eligible but did not participate in 
21° CCLC) were included (combined n = 10,242). Additionally, the effects of 21‘* CCLC 
participation on three subgroups, based on special education status, limited English proficiency 
status, and economically disadvantaged status, were examined. 

A further consideration is that the achievement and attendance findings can only be used 
to evaluate the performance of all centers in Virginia as a group, not the performance of any 
specific center, as the results were aggregated across all centers rather than evaluated center-by- 
center. 

Third-grade Only 

As most students in third-grade have no prior-year (i.e., pretest) test data available, it was 
not feasible to apply inferential statistics to these data because any statistically significant or 
substantively important differences between 21 CCLC participants (i.e., those with 30 or more 
days of attendance) and nonparticipants (i.e., eligible students with zero days of attendance) may 
not be the result of 21‘ CCLCs. Rather, differences could be the result of differences in prior 
ability, as it was not possible to either (a) determine if the participant and nonparticipant groups 
were similar on prior-year achievement, or (b) adjust 2017-2018 outcomes based on prior-year 
achievement for the third-grade students as SOL data are only available in grades 3 through 8. 

Consequently, separate descriptive (noninferential) analyses were conducted for 21“ 


CCLC participants and nonparticipants in grade three in 2017-2018, who had no prior-year test 
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data available. The analyses used the proficiency levels on the SOL assessments (based on the 
percentage scoring Proficient or Advanced) and mean (i.e., average) scaled scores on SOL 
assessments for 2017-2018 only. As a result, it would be more appropriate to use the findings 
from these analyses to better understand whether the program is serving students with an 
identified need (1.e., serving students on average who are the lowest achievers) vs. interpreting 
the findings as an evaluation of the effectiveness of the 21*t Century program. In other words, 
the outcomes should be used to learn more about the population being served rather than 
evaluating their outcomes. These analyses examined differences in reading and mathematics 
achievement between the following: 

(1) 21“ CCLC participant and nonparticipant third-grade students; 

(2) 21 CCLC participants and all Virginia third-grade students (where similar data were 
available). 

In addition to the comparisons between (a) all students in the 21‘' CCLC participant and 
nonparticipant groups, as well between (b) 21‘ CCLC participants and Virginia, comparisons 
between these three groups were also conducted by the following subgroups where common data 
were available: Gender, race, economic disadvantage status, disability status, and LEP status. 
The results for the grade-three-only analyses must be viewed as limited, as they are descriptive 
only; thus, it is possible that differences in achievement between participants and nonparticipants 
could be due to differences in areas such as prior ability or motivation, or due to chance, and may 
not be related to participation in the 21‘ CCLC program itself. Comparison data for Virginia 
were based upon the 2017-2018 State Report Card data from the Virginia Department of 


Education’s Web site at the following link: Virginia Department of Education School Quality 


Profiles. 
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Center and Participant Characteristics 
Center and participant characteristics were collected in the Annual Local Evaluation 

Report Template (ALERT) for both the regular school year and the summer. Almost all grantees 
(91%) participated in the online tool. However, since a few grantees did not participate, the data 
presented in the section on teachers’, students’, and parents’ participation does not represent the 
entire population. 
21%‘ CCLC Regular School Year Program 

Staff. There were 2,778 paid and volunteer staff members across the centers during 
2017-2018 academic year. Of these staff members, more than half (59%) were certified school- 
day teachers. Non-certified school staff, including but not limited to, paraprofessionals, 
librarians, trained tutors, cafeteria workers, and bus drivers, made up 31% of the centers’ staff. 
Additionally, approximately 11% of the staff were comprised of college and high school 
students, AmeriCorps volunteers, and community members who assisted as unpaid volunteers. 

Attendance by Grade Level. Approximately 10,000 students attended 21st CCLC 
programs regularly (i.e., 30 days or more) during the 2017-2018 school year. As shown in Table 
2, the majority of students served were in grades three through eight. The grade with the largest 
number of student participants was grade four (14%), followed closely by grades seven and five. 
Lower elementary (PreK — 2) and high school (9-12) had the smallest numbers of student 


participants. 
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Table 2. 218‘ CCLC Regular School Year Student Attendance by Grade Level 


Pre-kindergarten 31 0.3 
Kindergarten 239 2.4 
Ist grade 369 3.7 
2nd grade 658 6.6 
3rd grade 1270 12.7 
4th grade 1403 14.1 
5th grade 1329 13.3 
6th grade 1287 12.9 
7th grade 1342 13.5 
8th grade 1114 11.2 
9th grade 322 3.2 
10th grade 275 2.8 
11th grade 147 1.5 
12th grade 147 1.5 
Data not provided 35 0.7 
TOTAL 9986 100.0 


Demographics. Demographic information for students regularly attending 21st CCLC 
programs in 2017-2018 was also reported by grantees in the ALERT (See Table 3). Data reflect 
that 44% of students served by the 21st CCLC program are African American, 34% White, 14% 
Hispanic or Latino, 4% Two or More Races, 2% Asian, and less than 1% are Native Hawaiian or 
Pacific Islander or American Indian/Alaska Native. Less than 11% of students had limited 
English proficiency. Over three-fourths (78%) were eligible for free or reduced-price lunch, and 
nearly 15% of students had a special need or disability. About half of the 21st CCLC students 
were female. 

Compared to the state. When comparing the 21st CCLC student population to all the 
students served throughout the Commonwealth of Virginia for the 2017-2018 school year, the 
21st CCLC student population was representative of the Commonwealth in some particulars (See 
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Table 3). The ratio of females to males were comparable, as well as the number of limited 
English proficiency, special needs or disability, and Hispanic or Latinos served. However, there 
were differences when comparing the percentages of students eligible for free and reduced-price 
lunch, and the percentages of Whites and African Americans served. The 21st CCLC program 
had much higher numbers of students at an economic disadvantage and served a larger 


percentage of African Americans (Virginia Department of Education, 2017). 


Table 3. 21st CCLC and State Regular School Year Student Demographics 


Student Demographics CCLC Frequency CCLC Percentage mee mai 
Gender 
Female 5562 52.0 48.5 
Male 5115 47.8 51.5 
Data not provided 21 0.2 N/A 
Race 
American Indian or Alaska Native 29 0.3 0.3 
Asian 188 1.9 3.6 
Black or African American 4441 44.3 22.3 
Hispanic or Latino 1435 14.3 15.7 
Native Hawaiian or Pacific Islander 48 0.5 0.2 
White 3435 34.3 49.0 
Two or more races 426 4.2 5.6 
Data not provided 25 0.2 N/A 
Additional Information 
Limited English proficiency 1154 10.8 12.3 
Free or reduced-price lunch 8362 78.2 44.3 
Special needs or disability 1578 14.8 13.0 
Data not provided 116 1.1 N/A 


21%* CCLC Summer Program 
Staff. For centers that offered summer programs, 1,025 staff participated. Most of the 
summer staff (94%) were paid, and over half (57%) were certified school-day teachers. Of the 


64 volunteer summer staff members, 9 were certified school-day teachers, while the remaining 
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unpaid summer staff (7 = 55) included parents, high school students, paraprofessionals, retired 
teachers, and school administrators. 
Attendance by Grade Level. A total of 3,763 students attended 21st CCLC summer 


programs regularly (attending at least 75% of sessions) in 2017-2018. As in the regular school 


year, the majority of students were in grades three through eight. The largest group of 


participants were in grade six (15%), followed closely by grades four and three. Again, early 


elementary (PreK — 2) and high school (9-12) had the lowest student attendance numbers (See 


Table 4). 
Table 4. 218%' CCLC Summer Student Attendance by Grade Level 
Grade Level Frequency Percentage 

Pre-kindergarten 21 0.6 
Kindergarten 136 3.6 
Ist grade 152 4.0 
2nd grade 258 6.9 
3rd grade 535 14.2 
4th grade 540 14.4 
5th grade 494 13.1 
6th grade 546 14.5 
7th grade 445 11.8 
8th grade 366 9.7 
9th grade 108 2.9 
10th grade 55 1.5 
11th grade 56 1.5 
12th grade 45 1.2 
Data not provided 6 0.2 
TOTAL 3763 100.0 


Demographics. The demographic information of students regularly attending 2 1st 


CCLC summer programs in 2017-2018 was also collected. These demographic data reflect that 


55% of students served were African American, 30% White, 9% Hispanic or Latino, 4% Two or 


More Races, 1% Asian, and less than 1% were Native Hawaiian or Pacific Islander or American 
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Indian/Alaska Native. Less than 5% of students had limited English proficiency. Three-fourths 
(75%) were eligible for free or reduced-price lunch, and nearly 10% of students had a special 


need or disability. As seen in the regular school year, about half were female (See Table 5). 


Table 5. 21st CCLC Summer Student Demographics 
: (oe Ke (xe) Ke 
Student Demographics Frequency Percentage 


Gender 

Female 1902 50.7 
Male 1848 49.3 
Data not provided 0 0.0 
Race 

American Indian or Alaska Native 5 0.1 
Asian 38 1.0 
Black or African American 2027 54.7 
Hispanic or Latino 334 9.0 
Native Hawaiian or Pacific Islander 2 0.1 
White 1127 30.4 
Two or more races 136 3.7 
Data not provided 40 1.1 
Additional Information 

Limited English proficiency 178 4.7 
Free or reduced-price lunch 2803 74.7 
Special need or disability 341 9.1 
Data not provided 340 9.1 


21st CCLC Parents and Adult Family Members 

The 21st CCLC programs also served parents and adult family members of 21st CCLC 
students. Grantees reported a total of 4,263 parents and adult family members who attended 21st 
CCLC programs during the 2017-2018 school year, and 537 parents and adult family members 
who attended 21st CCLC summer programs. 
21st CCLC Activities 

A wide variety of activities were offered in 21*' CCLC centers across the state. Types of 


student activities reported in the ALERT included (a) academic remediation, (b) academic 
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enrichment, (c) tutoring, (d) homework help, (e) mentoring, (f) assistance for Limited English 
Proficient students, (g) recreation, (h) social skills instruction for students demonstrating 
inappropriate behavior (ex. truancy, violation of school rules), (i) career training, (j) expanded 
library hours, and (k) community service and youth leadership training. Most grantees’ overall 
goal for an activity was to “improve academic achievement in reading/English” or “improve 
academic achievement in Math”, as well “enrich opportunities for students.” However, quite a 
few focused on “providing character education.” 

While most activities were provided for students, about a fifth were provided for both 
“students and adult family members.” Activities for parents of the students enrolled in the 
CCLC program included (a) GED classes, (b) family nights at the center where parents worked 
with their children on projects, (c) volunteer opportunities at the school, (d) training in parenting 
skills, (e) computer classes, (f) English as a Second Language courses, and (g) community 
resource referrals. 

Results 

Grantees were required to address three objectives: 1) Improve student achievement in 
reading/language arts; 2) Improve student achievement in mathematics; and 3) Provide 
opportunities for parental education. The extent to which the centers met these objectives are 
presented below by objective. While not one of the statewide objectives, an analysis of school- 
day attendance was also included. Following that, aggregate results of the Promising Practices, 
the Student Perceptual Survey, and the Teacher APR Survey are provided. 

Objective 1: Improve Student Academic Achievement in Reading 
When looking at all matched 21*' CCLC participant and control group students in grades 


four through eight, after statistically controlling for student demographic variables and prior year 
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achievement, there was no statistically significant impact of 21‘ CCLC participation (“Yes” or 
“No”’) on 2017-2018 statewide reading assessment proficiency level or standardized SOL scaled- 
scores. Additionally, the magnitude of the effect size for the proficiency analysis (Cox Index 
effect size (CIES) = -0.01) and for the standardized SOL scaled score analysis (g = 0.00) were 
not substantively important (i.e., educationally meaningful) based on What Works Clearinghouse 
(2017) guidelines (i.e., => +/- 0.25) (What Works Clearinghouse, 2017). The effect size 
(calculated as either the Cox Index for the proficiency analyses or Hedges’ g for the standardized 
SOL scaled score analyses) is a descriptive statistic that provides a measure of the magnitude of 
the difference between scores (1.e., whether the difference is large enough to be meaningful, 
separate from whether the difference is statistically significant). 

There were no statistically significant or substantively important differences in reading 
proficiency levels for the limited English proficient or economically disadvantaged subgroups 
based on 21*t CCLC participation. However, students in 21‘ CCLC receiving special 
education (SPED) services statistically significantly outperformed SPED control students in 
reading proficiency level outcomes, although the effect was not substantively important (g = 
0.14). There was no statistically significant or substantively important difference in reading 
standardized SOL scaled scores for the economically disadvantaged subgroup based on 21* 
CCLC participation. When only looking at 21° CCLC participants, the number of days a student 
participated in 21‘' CCLC had no statistically significant or substantively important impact on 
either reading proficiency level or standardized SOL scaled score outcomes. 

The following student-level trends in statistically significant achievement outcomes 


emerged in reading over the past four years (2014-2015 to 2017-2018) (see Table 6 and Table 7). 
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For both 21 CCLC students only (Analysis of Center Effects) and for the 21*' CCLC vs. control 
students’ analyses 
e Non-special education students outperformed special education students on the 
proficiency analyses as well as the standardized SOL scaled score analyses. 
e Non-economically disadvantaged students outperformed economically disadvantaged 
students on the proficiency analyses as well as the standardized SOL scaled score analyses. 
For the 21 CCLC vs. control students’ analyses 
e Overall, there were no statistically significant impacts of participation in 21st CCLC on 
either proficiency level or standardized SOL scaled score measures of reading achievement. 
e Females outperformed males on the proficiency level and standardized SOL scaled score 
analyses. 
e Non-limited English proficient students outperformed limited English proficient students 
on the proficiency level and standardized SOL scaled score analyses. 
For the 21 CCLC only analyses 
e The number of days of participation in 21st CCLC had a statistically significant positive 
impact on both proficiency level and standardized SOL scaled score reading achievement in 
2016-17 only. 
e The impact of prior-year reading achievement was positive for both the proficiency level 
and standardized SOL scaled score outcomes, with higher achievement in the prior year 


translating into higher performance in the current year. 
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Table 6. Four-Year Achievement and Student-Level Demographic Summary of 
Statistically Significant Outcomes in Reading in Grades 4-8 for 215‘ CCLC Students 


Readin 


Cope 2014-2015 2015-2016 2016-2017 2017-2018 
Se Proficiency 
CCLC Only CCLC Only CCLC Only CCLC Only 
Student Demographics 
Number of CCLC 2 or 
ea Positive Positive 
participation days 
Heels Female Female Female Female Female 
higher higher higher higher higher 
Ww Ww WwW WwW WwW WwW WwW Ww 
White (reference group) higher higher Higher higher higher higher higher higher 
compared to Hispanic, than than than than than than than than 
ean can ane AA and AA and AA and AA and AA and 
Other race groups AA AA AA 
H H H H H 
Non- Non- Non- Non- Non- Non- Non- Non- 
Special Education Status SPED SPED SPED SPED SPED SPED SPED SPED 
higher higher higher higher higher higher higher higher 
Non- Non- Non- Non- 
Limited English Status LEP LEP LEP LEP 
higher higher higher higher 
Non- Non- Non- Non- Non- Non- Non- Non- 
Disadvantaged Status ED ED ED ED ED ED ED ED 
higher higher higher higher higher higher higher higher 
Prior Year Achievement Positive Positive Positive Positive Positive Positive Positive Positive 
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Table 7. Four-Year Achievement and Student-Level Demographic and Subgroup 
Summary of Statistically Significant Interaction Outcomes in Reading in Grades 4-8 for 
21%‘ CCLC vs Control Students 


Reading 


Coutane 2014-2015 RANTS 2016-2017 UAE: 
CCLC vs Control CCLC vs Control CCLC vs Control CCLC vs Control 
Student Demographics 
CCLC Participant 
Time Positive Positive Positive Positive Positive Positive Positive Positive 
Bewmalé Female Female Female Female Female Female Female Female 
higher higher higher higher higher higher higher higher 
WwW W WwW WwW WwW WwW W W 
White (reference group) higher higher higher higher higher higher higher higher 
compared to Hispanic, than than than than than than than than 
African American, and AA,O AA, H, AA, O AA AA AA AA AA 
Other race groups higher O higher higher and and and and and 
than W than W than W H H H H H 
Non- Non- Non- Non- Non- Non- Non- Non- 
Special Education Status SPED SPED SPED SPED SPED SPED SPED SPED 
higher higher higher higher higher higher higher higher 
Non- Non- Non- Non- Non- Non- Non- Non- 
Limited English Status LEP LEP LEP LEP LEP LEP LEP LEP 
higher higher higher higher higher higher higher higher 
cad vanmsed Status Non-ED Non-ED Non-ED Non-ED Non-ED Non-ED Non-ED Non-ED 
8 higher higher higher higher higher higher higher higher 
Interactions 
NA NA cae 
Special Education NA NA 
0.14 
NA NA 
Not Special Education ——_ >]. sp | NA PR NA 
na | Get [Na 
Limited English Proficient £ NA NA 
-0.21 
CCLC 
NA : NA 
Not Limited English higher 
Proficient Me ue 
0.06 
Economically 
Disadvantaged 
CCLC 
Not Economically higher 
Disadvantaged 0.11 


Note. The values below the group represent the effect size. 
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Results of the descriptive analysis of reading outcomes for students in grade three who 
did not have prior-year test scores available showed that for proficiency outcomes, 21*t CCLC 
participants in 2017-2018 were outperformed by non-participants in reading proficiency for all 
students combined and in 13 of 15 available subgroupings, and by Virginia in reading 
proficiency for all students combined and in 14 of 15 available subgroupings. In terms of SOL 
scaled scores, third-grade 218‘ CCLC participants in 2017-2018 were outperformed by non- 
participants overall and in 12 out of 15 subgroupings, and by Virginia for all students combined 
and in 14 of 15 subgroup comparisons in reading. 
Objective 2: Improve Student Academic Achievement in Mathematics 

When examining the combination of all matched control group and 218‘ CCLC 
participants in grades four through twelve, participation in 21S" CCLC programs (“Yes” or ““No”’) 
did not have a statistically significant effect on participants’ 2017-2018 statewide mathematics 
assessment proficiency level or standardized SOL scaled-scores, after controlling for student 
demographic variables and prior achievement. Additionally, the effect sizes for the proficiency 
level analysis (Cox Index effect size (CIES) = 0.04) and for the standardized SOL scaled score 
analysis (g = 0.03) were not substantively important based on What Works Clearinghouse (2017) 
guidelines (i.e., > +/- 0.25). There were no statistically significant or substantively important 
differences in mathematics proficiency level for the limited English proficient or economically 
disadvantaged subgroups based on 21*' CCLC participation. However, 21*' CCLC students 
receiving special education services statistically significantly outperformed control students 
receiving special education services in mathematics proficiency level outcomes, although the 
effect was not substantively important (g = 0.11). Meanwhile, on mathematics standardized SOL 


scaled scores, economically disadvantaged students in 21°‘ CCLC statistically significantly 
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outperformed economically disadvantaged control students, while non-economically 
disadvantaged control students outperformed 21*' CCLC non-economically disadvantaged 
students, but the magnitude of the effects were not substantively important (g = 0.05 and g = - 
0.08 respectively). When only looking at 21‘ CCLC participants, the number of days a student 
participated in 21‘ CCLC a statistically significant, but not substantively important, impact on 
mathematic proficiency level and mathematics standardized SOL scaled score outcomes. 

The following student-level trends in statistically significant achievement outcomes 
emerged in mathematics over the past four years (2014-2015 to 2017-2018) (see Table 8 and 
Table 9): 

For both 21 CCLC students only (Analysis of Center Effects) and 21° CCLC vs. control 
students’ analyses 

e Non-special education students outperformed special education students on the 
proficiency level analyses as well as the standardized SOL scaled score analyses. 

e Non-economically disadvantaged students outperformed economically disadvantaged 
students on the proficiency level analyses as well as the standardized SOL scaled score 
analyses. 

For the 21 CCLC vs. control students’ analyses 

e There were no statistically significant impacts of participation in 21st CCLC on either 
proficiency level or standardized SOL scaled score measures of reading achievement over 
the past four years. 


e Non-limited English proficient students outperformed limited English proficient students 
on the proficiency level and standardized SOL scaled score analyses. 
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For the 21 CCLC only analyses 


e The number of days of participation in 21st CCLC had a statistically significant positive 
impact on mathematics proficiency level achievement, as well as standardized SOL scaled 
scores measures of mathematics achievement for three out of four years. 

e The impact of prior-year mathematics achievement was positive for both the proficiency 
level and standardized SOL scaled score outcomes, with higher achievement in the prior 
year translating into higher performance in the current year. 

Table 8. Four-Year Achievement and Student-Level Demographic Summary of 


Statistically Significant Outcomes in Mathematics in Grades 4-8 for 215‘ CCLC Students 


Mathematics 


2014-2015 2015-2016 2016-2017 


; 2017-2018 
rate Proficiency | SOL 
CCLC Only CCLC Only CCLC Only CCLC Only 
Student Demographics 
Ded a clays of Positive Positive Positive Positive Positive Positive Positive 
participation in CCLC 
ne Female Female Female Female Female Female 
higher higher higher higher higher higher 
W W WwW WwW WwW W W W 
White (reference group) higher higher higher higher higher higher higher higher 
compared to Hispanic, than than than than than than than than 
African American, and AA AA AA AA 
Other race groups AA AA and and AA AA and and 
H H H H 
Non- Non- Non- Non- Non- Non- Nous Non- 
Special Education Status SPED SPED SPED SPED SPED SPED PED higher SPED 
higher higher higher higher higher higher . higher 
Non- Non- Non- Non- Non- 
Limited English Status LEP LEP LEP LEP LEP 
higher higher higher higher higher 
Non- Non- Non- Non- Non- Non- Non- Non- 
Disadvantaged Status ED ED ED ED ED ED ED ED 
higher higher higher higher higher higher higher higher 
Prior Year Achievement Positive Positive Positive Positive Positive Positive Positive Positive 
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Table 9. Four-Year Achievement and Student-Level Demographic and Subgroup 
Summary of Statistically Significant Interaction Outcomes in Mathematics in Grades 4-8 
for 21° CCLC vs Control Students 


Mathematics 
Cousins 2014-2015 2015-2016 2016-2017 2017-2017 
SOL 
(OO) OR Coyatsx0) CCLC vs. Control CCLC vs. Control CCLC vs. Control 
Student Demographics 
CCLC Participant 
Time Positive Positive Positive Positive Positive Positive Positive Positive 
Howat Female Female Female Female Female 
ae higher higher higher higher higher 
Ww WwW WwW Ww Ww WwW WwW WwW 
Minion Whiletekrcice higher higher higher higher higher higher higher higher 
anoup) compared to Hispanic than than than than than than than than 
African American, and Other ae Hh . ms ge aa AA, H, AA AA AA 
lied ci higher higher Higher higher i 
than W than W than W than W 
Non- Non- Non- Non- Non- Non- Non- Non- 
Special Education Status SPED SPED SPED SPED SPED SPED SPED SPED 
higher higher higher higher higher higher higher higher 
Non- Non- Non- Non- Non- Non- Non- Non- 
Limited English Status LEP LEP LEP LEP LEP LEP LEP LEP 
higher higher higher higher higher higher higher higher 
Non- Non- Non- Non- Non- Non- Non- Non- 
Disadvantaged Status ED ED ED ED ED ED ED ED 
higher higher higher higher higher higher higher higher 
Interactions 
wa | CLC | Na ccLC 
Special Education igner NA igher NA 
0.24 0.11 
NA NA 
Not Special Education NA NA 
NA ey NA 
Limited English Proficient poo ef NA NA 
-0.12 
CCLC 
Not Limited English NA higher NA NA NA 
Proficient 0.06 
CCLC CCLC 
high high 
Economically Disadvantaged _ _ 
.04 0.05 
Control 
Not Economically higher 
Disadvantaged -0.08 
Note. The values below the group represent the effect size. 
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The results of the grade-three-only analyses of mathematics proficiency level data 
showed that third-grade 21%‘ CCLC participants in 2017-2018 outperformed non-participants 
on 10 out of 15 subgroupings. However, Virginia third-graders outperformed 21‘t CCLC third- 
graders in mathematics proficiency for all students combined and on 12 out of 15 subgroupings. 
In terms of mathematics SOL scaled scores, while third-grade non-participants did better for all 
students combined, 21% CCLC participants outperformed non-participants in 8 out of 15 
subgroups. In addition, Virginia third-graders did better for all students combined, as well as in 
13 out of 15 subgroups compared to 21*' CCLC participants in 2017-2018. 

Improve Student School-Day Attendance. When examining the 21*' CCLC participant 
and matched control group in grades four through twelve, after controlling for student 
demographic variables and prior year school-day attendance, participation in 21** CCLC 
programs (“Yes” or “No’”) had a statistically significant positive effect on participants’ 2017- 
2018 school-day attendance, although the effect size (g = 0.06) would not be considered 
substantively important, with CCLC students attending slightly more than one-quarter day more 
on average. In addition, there were statistically significant, but not substantively important, 
positive differences in school-day attendance (favoring 21‘ CCLC participants) for the not 
limited English proficient (g = 0.06), special education (g = 0.10), not special education (g = 
0.05), and the economically disadvantaged (g = 0.07) subgroups based on 21*t CCLC 
participation. 

Objective 3: Provide Opportunities for Parental Education 

Grantees developed and provided a variety of programs and activities based on seven 

subobjectives: (a) General Education Development (GED) certificate program, (b) computer 


skills instruction, (c) parenting skills classes, (d) opportunities for parent/child interaction, (e) 
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career development information/training, (f) health and wellness training, and (g) “Other” 
subobjectives and activities. The percentage of centers selecting each subobjective is shown 
below in Table 10. The results for each subobjective are described following the table. 


Table 10. Parent Education Subobjectives Selected by Grantees 
You selected provide parent education as an objective. 


Indicate the subobjective(s) you selected for your center's program. Percentage 
(Check all that apply.) 


Opportunities for parent/child interaction in academic activities 83.9 
Parenting skills training 50.9 
Health and wellness training 41.1 
Career development information/training 25.0 
Computer skills instruction 22.3 
General Educational Development (GED) certificate program 13.4 
Other subobjectives and activities 11.6 


Parent/Child Interaction in Academic Activities. Opportunities for parent/child 
interaction in academic activities was the subobjective selected most commonly (84%) by 
grantees. Centers that offered this opportunity reported using sign-in sheets, attendance logs, and 
parent surveys to determine whether they had met this subobjective. These opportunities were 
offered through schools, centers, and community partners. This subobjective was reported to be 
met by nearly all (97%) of the centers providing opportunities for parent/child interaction in 
academic activities. 

Parenting Skills Training. Providing parenting skills training was a subobjective 
selected by over half (51%) of the centers, and a variety of data sources were used to determine 
whether they had met this subobjective, including participation sign-in sheets, parent surveys and 


interviews, as well as registration records. Parenting skills training was often reported to be 
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offered via center programs and family events. Again, nearly all (97%) of the centers providing 
parenting skills training reported meeting this subobjective. 

Health and Wellness Training. Health and wellness training was selected as a 
subobjective by 41% of reporting centers, and they utilized sign-in sheets, attendance logs, 
meeting logs, interviews, and parent surveys as sources to determine whether they had met this 
subobjective. A variety of opportunities was offered at centers, schools, or during family 
programs. Almost all (96%) of the centers providing health and wellness training reported 
meeting this subobjective. 

Career Development Information/Training. This subobjective was selected by a 
quarter (25%) of the reporting centers. Career development information/training opportunities 
were made available to parents and family members at centers, schools, community partners, 
community colleges, and institutions of higher education. Centers that reported career 
development opportunities used a variety of data sources to determine whether they had met this 
subobjective, including attendance records, parent surveys, meeting logs, and sign-in sheets. 
Nearly all (96%) of the centers providing career development information/training reported 
meeting this subobjective. 

Computer Skills Instruction. Over a fifth (22%) of the reporting centers selected parent 
computer skills instruction as a subobjective. Centers that provided computer instruction and 
activities used a variety of measures to determine that they had met this subobjective, including 
sign-in sheets, participation logs, and completion forms, as well as parent surveys and 
interviews. Centers reported offering parents and family members free access to online courses 
covering topics such as computer skills and career readiness, as well as outlining online 


resources for reviewing their children’s important academic information (e.g., grades, 
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attendance, assignments, and SOL test scores). A very large percentage (92%) of the centers 
providing computer skills instruction reported meeting this subobjective. 

General Education Development. General Education Development (GED) certification 
was selected as a subobjective by 13% of the reporting centers. Some GED certificate program 
classes were scheduled at the center, while others were offered by GED certification programs in 
the community. While several grantees reported GED graduates during the 2017-2018 school 
year, a few grantees indicated that some parents/family members had inquired about the GED 
program, completed initial screenings, or were still working towards completion of the GED 
requirements. Approximately half (53%) of the centers providing GED certification reported 
meeting this subobjective. 

Other Subobjective Activities. A few of the reporting centers (12%) selected other 
subobjective activities and used attendance reports, interviews, and parent surveys to determine 
whether they had met this subobjective. Centers reported offering parents and family members 
English Language Learner classes, as well as sessions on business management and tax 
preparation. Many of the centers providing other subobjective activities (88%) reported meeting 
this subobjective. 

A summary of the success of parent education subobjectives is provided below in 
Table 11. Again, it is important to note that grantees determined their own criteria for success in 


meeting parental education objectives and reported their outcomes accordingly. 
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Table 11. Success in Meeting Parent Education Subobjectives 


% of 
Parent Education Subobjective oleae es % Yes % No 

selected this 

subobjective 
Did parents participate in the GED certificate program classes? 13.4 53.3 46.7 
Did parents participate in computer skills classes? 22.3 92.0 8.0 
Did parents participate in parenting skills training classes? 50.9 96.5 35 
Did parents and children have opportunities to interact in academic 83.9 96.8 32 
activities? 
Did parents participate in career development information/training 25.0 96.4 0.0 
sessions? 
Did parents participate in health and wellness training sessions? 41.1 95.7 4.3 
Did parents attend the "Other" subobjective activities? 11.6 92.3 0.0 


Note. Item percentages may not total 100% because of missing input from some respondents. 
p gS ly g§ inp p 


Association Between Center Characteristics and Outcomes 

This section of the evaluation report includes the results of statistical analyses of 
associations between various categories of center-level data and reading and mathematics 
outcomes of students in grades four through twelve participating in 21‘‘ CCLC with two years of 
assessment data available. These analyses provide information that may be useful to program 
leaders and are summarized below. 

Table 12 shows the trends for center activities over the last 10 years. From 2008-2009 to 
2011-2012, there was a decrease each year in the total number of individual activities that the 
centers have offered. For 2015-2016, the mean number of activities decreased to the lowest 
level and decreased each year from 2009-2010 to 2015-2016. For the last three years (2015- 
2016 to 2017-2018) the total number of individual activities, as well as the mean number of 
activities has increased each year. In 2017-2018, the total number of center activities was at 
the second highest level, while the mean number of center activities was at the highest level. Jt 
should be noted that the way the data on activities were collected changed in 2015-2016, and the 
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number of activities was no longer differentiated between the school year and summer as they 
had been in previous years. There was also an upward trend in the total number of providers 
from the 2015-2016 to the 2017-2018 year, with 2017-2018 having the second highest number of 
providers. The 2008-2009 year had the highest total number of activities (7 = 1,265) and the 
largest number of providers (n = 144). The 2017-2018 year had the highest mean number of 
activities (n = 21.0) and the second highest total number of activities (7 = 1,241), along with the 
second highest number of providers (n = 118). The 2011-2012 year had the lowest total number 


of activities (n = 737) and the lowest number of providers (7 = 63). 


Table 12. Center Activities by Year 
Total Number Mean Number Number 


of Activities of Activities of Providers 
2017-2018 1,241 21.03 118 
2016-2017 1,079 9.1 114 
2015-2016 914 8.7 101 
2014-2015 1,001 9.7 112 
2013-2014 1,123 10.1 101 
2012-2013 981 10.8 71 
2011-2012 737 11.7 63 
2010-2011 928 12.5 74 
2009-2010 1,155 13.2 87 
2008-2009 1,265 8.8 144 


Association Between Center Characteristics and Reading Achievement 
The total number of staff at the center was a statistically significant positive predictor 


of both proficiency level and standardized reading SOL scaled scores. However, the impact was 
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negligible. For each additional increase in the number of staff, there was a 1.2 percent increase 
in the odds of scoring proficient and a 0.002 increase in reading standardized SOL scaled scores. 

A separate set of reading analyses for students with (a) one to 29 days of attendance in 
218 CCLC, (b) 30 to 59 days of attendance in 21*' CCLC, (c) 60 or more days of attendance, and 
(d) for the combined group of all students with one or more days of attendance in 218 CCLC 
were also conducted. Results of the correlation analyses revealed that for the group of students 
who attended 21‘ CCLC for 1-29 days as well as for the combined group of students who 
attended 21‘ CCLC more than one day, there was a statistically significant negative correlation 
between days attended and 2017-2018 reading standardized SOL scaled scores, with more days 
of attendance being associated with a decrease in reading standardized SOL scaled scores. 
However, the magnitude of the relationship (7 = - 0.02 and 7 = - 0.06 respectively) is considered 
very small, with attendance explaining 0.04 and 0.36 percent of the variation in reading 
standardized SOL scaled scores respectively. There was no statistically significant correlation 
between days attended and 2017-2018 reading standardized SOL scaled scores for students who 
attended 21‘ CCLC either 30 to 59 days or 60 days or more. 

None of the center-level variables included had a consistently statistically significant 
impact on achievement outcomes in reading over the past four years (2014-2015 to 2017-2018) 


for the 21‘' CCLC students only analyses (Analysis of Center Effects) (see Table 13). 
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Table 13. Four-Year Achievement and Center-Level Outcomes Summary of Statistically 
Significant Outcomes in Reading for Grades 4-8 


2014-2015 2015-2016 2016-2017 AVE ANE: 


Proficiency | SOL | Proficiency SOL 
CCLC Only CCLC Only CCLC Only CCLC Only 


Center Information 


Total number of hours 
of activities at the 
center 


Total number of 
activities at the center! 


Percent of activities Positive 
that were mathematics 
related! 0.002 


Percent of center 
activities that were 
English/reading 
related! 


Negative Positive 


-0.001 1.009 


Percent of certified Positive Positive 
paid school-day 
teachers at the center 1.006 0.001 


Positive Positive Positive 


Total number of staff 
0.001 1.01 0.002 


Percent of center Positive 
activities that were 
academic 1.006 


Percent of center 
activities that were 
enrichment! 


Note. The values for the group represent the coefficient for the analysis. 
‘Variable first measured in 2015-2016 


Association Between Center Characteristics and Mathematics Achievement 

The number of days attended was shown to be a negligible, but statistically significant 
positive predictor of mathematics proficiency level and mathematics standardized SOL scaled 
score outcomes. For each additional day of participation, there was a 0.001 increase in 


mathematics standardized SOL scaled scores and a 0.4 percent increase in the odds of scoring 
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proficient. In other words, there would need to be an increase of 1,000 days of participation in 
21“ CCLC to get a one scaled score point increase in mathematics SOL scores. No center-level 
variables were statistically significant predictors of mathematics proficiency level or 
mathematics standardized SOL scaled score outcomes. 

A separate set of mathematics analyses for students with (a) one to 29 days of attendance 
in 21‘ CCLC, (b) 30 to 59 days of attendance in 21‘ CCLC, (c) 60 or more days of attendance, 
and (d) for the combined group of all students with one or more days of attendance in 21** CCLC 
were also conducted. Results of the correlation analyses revealed that for the combined group of 
students who attended 21‘ CCLC for one day or more, there was a statistically significant 
negative correlation between days attended and 2017-2018 mathematics standardized SOL 
scaled scores, with more days of attendance being associated with a decrease in the mathematics 
standardized SOL scaled scores. However, the magnitude of the relationship (7 = - 0.03) was 
considered very small, with attendance explaining 0.09 percent of the variation in mathematics 
standardized SOL scaled scores. There was no statistically significant correlation between days 
attended and 2017-2018 mathematics standardized SOL scaled scores for students who attended 
either 1 to 29 days or 30 to 59 days. There was however, a statistically significant positive 
correlation between days attended and mathematics standardized SOL scaled scores for students 
who attended 21*t CCLC for 60 days or more (r = 0.05), although again, the correlation was very 
small, with attendance explaining 0.25 percent of the variation in mathematics standardized SOL 
scaled scores. 

None of the center-level variables included had a consistently statistically significant 
impact on achievement outcomes in mathematics over the past four years (2014-2015 to 2017- 


2018) for the 21*‘ CCLC students only analyses (Analysis of Center Effects) (see Table 14). 
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Table 14. Four-Year Achievement and Center-Level Outcomes Summary in Mathematics 
for Grades 4-12 


Mathematics 
2014-2015 2015-2016 2016-2017 WEAK 
CCLC Only CCLC Only CCLC Only CCLC Only 
Center Information 
Negative 

Total number of hours of 

tiviti t th t 
activities at the center 0.001 
Total number of activities 
at the center! 
Percent of activities that 
were mathematics related! 
Percent of center activities Negative Positive 
that were English/reading 
related! -0.002 1.011 
Percent of certified paid 
school-day teachers at the 
center 

Positive 
Total number of staff 
0.002 
oe Positive Positive 

Percent of center activities 
that demi 

at were academic 1.009 0.002 

é aoe Positive 

Percent of center activities 
that ichment! 

at were enrichmen 1.008 


Note. The values for the group represent the coefficient for the analysis. 
'Variable first measured in 2015-2016 


Promising Practices and Recommendations 

Hundreds of promising practices that were most effective in helping the grantees meet 
their objectives were reported in the ALERT. The most frequently-mentioned practices are 
discussed below, presented in order of the open-ended questions they address, and organized by 


themes; then, followed by a discussion of the recommendations’ most prominent themes. 
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1) What activities or promising practices appeared to be most effective in helping to 
meet your subobjectives for improving student academic achievement in math? 

2) What activities or promising practices appeared to be most effective in helping to 
meet your subobjectives for improving student academic achievement in 
reading/English? 

Note: The grantees’ responses regarding promising practices for improving achievement in math 
(QI) and in reading/English (Q2) were to a great extent duplicative. Therefore, these responses 
are considered below in a single discussion. 

In grantees’ responses to the questions about promising practices in both math and 
reading/English (English), the same three broad themes appeared most prominently. In order of 
frequency, these were (a) direct academic support, (b) program staff and structure, and (c) 
enrichment activities. In both sets of responses, the largest number of promising practices were 
those aimed at support for the students’ academic needs. Among these practices, the two most 
commonly mentioned were tutoring and homework help, with respondents frequently 
mentioning them together. 

Overall, homework help and tutoring were seen uniformly as very effective and 
fundamentally important. As one grantee said in responding to the question on promising math 
practices, “Tutoring played a monumental role in improving students’ academic achievement in 
math.” Another noted that daily teacher availability for these activities “helped students gain 
positive math achievement through consistency and immediate feedback.” A third respondent 
observed that, in addition to improving in math, students actually requested more tutoring. 

Tutoring and homework sessions were conducted either in small groups or one-on-one. 


Comments regarding promising practices in English described individualized or differentiated 
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tutoring as most effective. In comments discussing instructional methods, both math and English 
respondents emphasized the use of project-based learning (PBL) and active/interactive learning 
methods as effective practices. Math commenters named practices that involved the use of art, 
music, and movement to help with math memorization skills. In PBL, students were seen as 
“more receptive” to learning math objectives, and a grantee commenting on promising practices 
in English saw PBL as the “most effective” strategy because students “truly bought into the 
premise.” This individual also said students “didn’t know they were learning, and enjoyed the 
process of creating.” Dividing students by reading levels rather than by grade levels was an 
additional effective practice employed for English. Tutoring in English often was described as 
“standards-based”, or as being focused on passing the Standards of Learning (SOL) tests. 
Another respondent to the question on practices in English described using regular feedback 
from school-day teachers, as well as student assessment scores, in order to tailor instruction to 
student needs. 

In addition to instructor-led academics, substantial contingents of both math and English 
respondents reported using online or computer-based instructional programs, including 
interactive and game formats. Both groups reported using DimensionU, iReady, Stride Academy, 
and Study Island. Both math and English versions of the Sumdog and [XL programs were also 
used. In addition, math students were reported to use Dreambox, while English participants used 
Kahoot, Newsela, and Storybird. Also used for English was WriteBrain, a program for writing 
enrichment in which children author and “publish” their own books. 

Other important promising practices related to the make-up of the centers’ teaching staff. 
Both math- and English-related responses emphasized the desirability of having tutors who were 


qualified and certified school-day core subject teachers, preferably at the students’ grade level 
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and ideally the students’ own school-day teacher(s). At least one center employed students’ 
homeroom teachers. In addition to these professional teachers, some commenters on math 
practices mentioned as a promising practice the use of peer tutors, while respondents describing 
promising practices in English mentioned volunteer college and university students and 
Americorps participants as tutors. In English, some centers had tutors specializing in ELL 
services, and one grantee described having a “literacy specialist” working individually with the 
students. Both sets of respondents strongly emphasized enhanced communication and 
coordination between afterschool and day school staff as a promising practice. 

Numerous responses regarding both math and English practices described enrichment 
activities, but discussion of enrichment is reserved for Question 5, which specifically addresses 
this domain. 

3) What activities or promising practices appeared to be most effective in helping to 
meet your subobjectives for providing parent education? 

The most prominent theme among promising practices for parent education was 
workshops and other programs providing educational sessions for parents. This was followed in 
emphasis by a variety of family events and outreach, communication, and relationship building 
with families. 

Among the educational sessions most prominently mentioned were programs to help 
parents gain access to or use technology through computer labs and educational websites, as well 
as computer and internet literacy training; and to provide online resources to assist parents in 
helping their children at home. Educational websites mentioned included Splash Math, Lexis, 
Dream Box, Ten Marks, Study Island, IXL, and SOL Practice. Also described as effective were 


GED preparation classes and college and career information, including workshops on professions 
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and how to gain admission to them. Literacy and adult English classes, as well as ELL courses, 
were described as “most promising,” along with financial literacy and physical and mental 
wellness training. Other offerings included Yoga and mindfulness classes. 

In addition to onsite events, field trips involving parents were seen as very engaging and 
effective. The importance of community partners in providing parent education was stressed 
frequently. These partners included universities or colleges, community colleges, a housing 
authority and other local governmental entities, and local businesses. 

Effective elements of program structure that enhanced parent engagement and attendance 
included programming that the students planned or led, which allowed students to display 
projects or learned skills, and that involved interactions among parents or between parents and 
students. An example named by one grantee was a “student showcase linking school day 
learning to afterschool.” 

Incentives such as the provision of meals or snacks and the provision of transportation or 
childcare were described as important to increase parental engagement and attendance. Other 
incentives mentioned by grantees included giveaways of books and other donated items such as 
gift cards. In addition, flexible scheduling and the practice of surveying parents to determine 
their interests and what they saw as convenient times for activities were said to be promising 
practices. 

Emphasized almost as much as the content and structure of events and sessions were 
outreach to, and communication and relationship building with, the families of children in the 
program. Grantees repeatedly stressed the importance of using a variety of communication 
methods with parents such as: “callbacks and active listening,” flyers, newsletters, calendars of 


events and activities, and invitations with RSVPs sent home with students and by mail and 
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telephone. It was also emphasized that eliciting parent feedback and suggestions was a promising 
practice. At least one center was translating all communications into Spanish and using bi-lingual 
staff for telephone communications. Another grantee described having a Family Resource 
Coordinator who allowed the program to focus on parents and families with invitations, 
reminders of events, resources, and who also served as a personal contact to whom parents could 
communicate their concerns. 

4) What activities or promising practices appeared to be most effective in helping to 

meet your subobjectives for providing character education? 

Most of the comments submitted by respondents to this question described the 
instructional methods, activities, and techniques used by grantees in character education (CE). 
This was followed in frequency by references to the partners who worked with the program in 
this area. 

The overarching theme regarding the practice of character education seemed to be the 
idea of building CE into as many activities as possible, and successfully using these activities to 
improve behavior. As one site coordinator described it, “character education is woven throughout 
the school day.” The intent of this was to create positive school climate and “instill critical 
academic, social, [and] emotional skills and core character traits.” Ideally, the lessons derived by 
the students would be coordinated with disciplinary policy. One respondent said, “We refer to 
our pillars of character [Pillars of Character are elements of Character Counts, a CE training 
program] during any discipline incidents.” Also related to improved behavior, one grantee listed 
as promising practices, ““Addressing needs as they occur and listening to signals. Being proactive 


when the signals happen, and being timely about the response.” 
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The subject matter of CE and promising methods used varied greatly among centers. The 
subject matter of workshops, activities, and training included, among other things: substance 
abuse prevention; anti-bullying training; violence prevention; and a “buddy system” whereby 
students learned to use positive affirmations with one another and their teachers. At one center 
offering the “Game Changers” violence prevention program, volunteer facilitators were 
described as building “ongoing relationships with students and teachers,” and the site coordinator 
was counseling parents and students together “to problem solve [sic] conflicts and develop action 
plans for success.” The most promising real-world activities included community service work, 
which was seen as very effective in teaching the students compassion. 

A grantee related that, as seen by his/her program, effective CE was a two-step process. 
First, skills were learned in CE training, sometimes through direct instruction, which included 
“knowing what it looks like to have strong character, build good relationships, and attaining the 
soft skills to be successful in daily life.” Second, these skills were put into practice “organically” 
by the students while working in PBL projects. Another grantee said that students were “more 
receptive” in learning CE when provided with PBL and found it more effective to “integrated 
character education” than teaching it in a stand-alone lesson. 

In addition to learned skills, respondents described rewards and motivators which were 
seen as improving both program attendance and desired behaviors. They described incentives 
such as gift cards, “student of the week” awards, and “club cash,” which could be redeemed to 
“purchase” items at a club store. In addition, it was noted that field trips were seen as effective 
motivators for student attendance. 

Family and parent participation also were mentioned as elements of successful CE. One 


grantee described a holistic approach engaging all the adults in the students’ lives: “We strove to 
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work with all the adults in our students’ lives (parents, teachers, staff, and counselors) to ensure 
the students in our program always felt supported and were given the resources to truly succeed. 
We always highlighted good behavior through positive reinforcement, student of the week, and 

positive phone calls home.” 

The grantees described collaborations on CE with a large number of community partners 
and vendors as promising practices. These ranged from a county sheriff's office, a military base, 
and New River Valley Community Services; to programs like Project Discovery and 
Communities in Schools; and agencies like the Boys & Girls Club and the Boy Scouts. 
Specifically, mentoring for the students came from a number of sources, including Virginia Tech 
University students (both mentoring and tutoring); mentors provided by other community 
partners; peer mentors; and mentor clubs known as the Sisterhood Club and the Brotherhood 
Club. One grantee said that students were most responsive to activities that “allowed time to 
interact with partners who share life lessons and offer opportunities for students to share 
information about and examine themselves to focus on ways to achieve their goals in life.” 

Experienced and dedicated staff who were invested in the students and could serve as 
good role models were also described as very important in developing CE. Grantees said that 
their staff received professional development aimed at training them to model good character. A 
few also described an interesting communication technique wherein students wrote poetry to 
help staff “understand past incidents in their lives.” 

5) What activities or promising practices appeared to be most effective in helping to 
meet your subobjectives for providing enrichment opportunities? 

The most heavily emphasized theme found in grantees responses about enrichment was 


the wide variety of activities and trainings the centers were providing. Giving students a choice 
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in the selection of activities and the rotation of students through various activities were 
frequently mentioned practices that enhanced this variety. 

The most prominent general category of activity was field trips, which served numerous 
purposes. The excursions might be to a business, college or university, or to clubs in which 
students participated. They could be related to books the students had read, intended to enhance 
SOL learning, or related to specific areas of knowledge (e.g., trips to the Marine Science 
Museum or the Air & Space Museum). Grantees pointed out that field trips were “most 
effective” in providing enrichment because they offered hands-on experiences that would not 
otherwise be available to the students. One respondent put it more directly as applied to his/her 
constituent students, saying that field trips exposed students “to the world outside their 
immediate neighborhoods,” and also that they correlated to Virginia SOL, broadening horizons 
and making the dream of college appear attainable. There was a plethora of specific activities 
and classes named, including such domains as crafts, art, theater, clubs, college and career- 
focused experiences, gardening, real-world math problems, JAM Kids, MicroSociety, 
photography, yoga, and dance. 

STEM and STEAM subject matter were stressed as topics for enrichment, and PBL was 
again named as a most effective instructional method. 

6) What activities or promising practices appeared to be most effective in helping to 
meet your subobjectives for improving community partnerships? 

Responses to this question were overwhelmingly about communication and outreach 
among program staff and partners. Thus, promising practices included establishing effective 
communications with partners by all methods. Grantees responded that communications and 


collaboration should be regular, open, continuous (year-round), two-way, and should include 
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face-to-face meetings. The respondents stressed that, in addition to keeping partners informed, 
program coordinators should invite them to events and activities. When communications were 
open, grantees reported that they found partners to be eager to provide them with ideas, 
suggestions, and support. 

Other promising practices were to “reach out to supervisors and staff’ when working 


with a new partner and to bring in a variety of partners to address the students’ individual 


interests. It was suggested that grantees should ask their communities for expertise and guidance, 


and that they should seek “fresh ideas” from their students. Examples of partner contributions 


beyond typical support or enrichment activities were the provision of food for backpack 


programs by college students and Americorps members, and a sheriff’s department which wrote 


a grant to allow employment of a school resource officer. In a cautionary note, one grantee 
warned against allowing partner activities to interfere with attendance at a center’s remedial 
activities. 

Additionally, the “flexible and responsive” scheduling of partner events and meetings 
was described as a promising practice, as was a focus on “things that are interesting to both 
students and partners.” Finally, a practice that had proved effective was conducting pre- and 
post- surveys or debriefing sessions regarding activities, thus maintaining a “continuous 
improvement mind-set” among partners and staff. 

7) What activities or promising practices appeared to be most effective in helping to 
meet your “other” subobjectives? 

Very few grantees responded to this open-ended question. One grantee did specify that 
their other objective was attendance, while the rest described a variety of promising practices. 


The respondent who specified attendance as their other objective had enhanced program 
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attendance by announcing some enrichment activities ahead of time to ensure that “they will be 
at school so they can attend the program that day.” 

Other promising practices for which the objectives can only be inferred included 
incorporating two new partners for a program; starting a new 4th and Sth grade science program 
involving hands-on learning activities that students “really loved” aimed at preparing the 
students for 5th grade Science SOLs; expanding cooking classes by using a “train the trainer” 
model that allowed the grantee to hold more sessions; adding a Teen Cuisine curriculum; and 
getting support from administration with continuous partner and staff communications. Finally, 
one grantee simply named “projects” and a cooking club. 

8) Provide recommendations that might improve the program in the future. 

Several hundred recommendations were also submitted by grantees. Most had been 
mentioned earlier by grantees as “promising practices.” Among ideas not emphasized earlier 
were several under the general theme of “data.” These included adding time to the site 
coordinator’s schedule for data analysis; developing a data retrieval system “to enable 
longitudinal analysis of the grant in year 2;” greater consistency between state and federal 
reporting requirements; and larger submission windows for both state and federal reporting. It 
was also suggested that students be given pre-and post-assessments when attending field trips. 

Other recommendations were related to grant-writing and coordination, as well as 
understanding grants. There was a recommendation of professional development for teachers on 
grant requirements. It was also suggested that there should be a workshop on writing new grants 
after Year 3 ended; training and information for first-year grant coordinators; and longer grant 


cycles. 
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Furthermore, there were a number of recommendations for increased resources. Among 
these were “increased resources” for math and English tutors and a list of “pre-approved 
resources” for student activities and enrichment, including money for admission to arts and 
music events. It was suggested that “parent engagement interventions” should include enhanced 
GED support, more workshops on college/career themes, and in general more activities to 
enhance the employability of the parents. One grantee recommended once-a-month “parenting 
wisely” lessons. Finally, there were a number of recommendations for more student 
“ownership,” including more input from a student leadership committee; more student choice in 
projects, enrichment, and field trips; more student-led events; and allowing students to set 
academic goals. 

Student Perceptual Survey 

The survey was completed by 6,055 students, of whom 49% attended elementary school, 
42% attended middle school, 8% attended high school, and 1% did not respond to this question 
or exited the survey. 

Overall, the results indicated that students held a positive perception of their 21st CCLC 
program. When asked why they attended the 21st CCLC afterschool programs, the most 
commonly chosen responses were, “The afterschool program helps me do better in school” 
(59%), “the activities are fun” (55%), and “I like going to the afterschool program” (54%). Still, 
quite a few went to the afterschool program because their teacher wanted them to go (17%) or 
there was nothing else to do (20%). The results are summarized in Table 15 from highest to 


lowest percentage. 
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Table 15. Why Students Attend 215%* CCLC Programs 
I go to the afterschool program because (mark all that apply): Percentage 


The afterschool program helps me do better in school. 59.1 
The activities are fun. 54.7 
I like going to the afterschool program. 53.8 
My friends go. 40.3 
My parents want me to go. 38.3 
There is nothing else to do after school. 20.0 
My teacher wants me to go. 16.8 


When asked questions about the general impression of the afterschool program, most 
students indicated that they “feel safe” during 21st CCLC programs (78%), “the staff encourages 
them to do their best” (74%), they trust that “afterschool program will help them to learn the 
knowledge and skills that will help in school or college” (74%), and “they trust the staff’ (74%). 


Students’ overall perceptions of the 21st CCLC program are presented below in Table 16. 


Table 16. 218‘ CCLC Students’ Overall Perceptions of the Program 


I feel safe. 77.6 17.2 3.3 
I get to choose the activities I want to do. 36.1 39.2 22.5 
There are a lot of different activities. 71.7 20.4 5.7 
I have time to finish my homework. 67.5 23.5 7.0 
I learn how to study for tests. 58.5 24.4 14.9 
I get help with my homework. 63.7 23.2 10.9 
I get along well with other students. 55.7 36.4 5.8 
There are too many rules. 27.5 24.2 45.6 
The staff encourage me to do my best. 74.4 17.2 6.0 
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During the afterschool program: Yes Sometimes No 
I trust the staff. 73.7 18.9 5.0 
The staff is nice. 64.8 28.5 4.1 
The staff listens to me when I have something to say. 62.4 28.6 5.9 
I learn skills that help me when I am not at school. 57.1 26.2 13.2 
I receive information that will help me find a job or choose a 64.0 23.9 10.5 
career. 
The afterschool program teaches me about professional behavior. 69.1 20.8 8.8 
The afterschool program helps me learn the knowledge and skills 70.6 20.6 16 
that I will need to be ready for a job or career. ; ; ; 
I receive information that will help me choose a trade school or 675 204 113 
college. 
I receive information or assistance with the trade school or college 384 193 373 
admissions process. 
The afterschool program helps me learn the knowledge and skills 743 16.5 78 


that I will need to be ready for trade school or college. 


Note. Item percentages may not total 100% due to multiple or missing responses by some students. 
P § yy Ip § resp 


Students were also asked if going to the 21st CCLC programs benefited different areas of 


their life (See Table 17). Most items garnered substantial agreement. The items with the highest 


agreement among 21st CCLC students included: “prepare for trade school or college” (73%), 


“prepare for a job or a career” (72%), “get better grades in school” (72%), and “be better at 


math” (71%). The items with the highest disagreement included: “enjoy school more” (16%), 


“make new friends” (12%), “be a better reader” (10%), and “attend class regularly” (10%). 
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Table 17. 218* CCLC Students’ Perceptions of Program Benefits 


Going to the afterschool program has helped me to: Disagree 

Be a better reader. 

Be better at math. 71.0 14.7 7.8 
Make new friends. 63.9 17.1 12.3 
Turn in my homework on time. 65.7 18.2 9.5 
Attend class regularly. 66.6 16.6 10.1 
Behave well in class. 65.8 19.4 8.2 
Get better grades in school. 72.0 15.7 5:7 
Get along well with other students. 60.4 23.9 8.9 
Enjoy school more. 56.9 19.9 16.0 
Prepare for a job or a career. 724 20.2 5.1 
Prepare for trade school or college. 73.0 17.1 7.6 


Note. Item percentages may not total 100% due to multiple or missing responses by some students. 
P E ly Ip 


Teacher APR Survey. There were 7,465 surveys completed by school-day teachers. 


Most of the respondents either taught mathematics and/or English to their 21st CCLC students. 


The majority of students (73%) were reported to be in grades three through eight and most 


students (89%) attended the programs for 30 days or more (Table 18, Table 19, and Table 20). 


Table 18. Subject taught to 2017-2018 21%* CCLC Students 


English 41.8 
Mathematics 34.2 
Science 12.5 
History/Social Science 8.0 

Note. Item percentages may not total 100% because of missing input from some respondents. 

Table 19. 2017-2018 21%' CCLC Students’ Grade Level 

Student grade level Frequency | Percentage 

Pre-K 22 0.3 
Kindergarten 219 2.9 
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Student grade level Frequency | Percentage 


Ist grade 303 4.1 
2nd grade 527 71 
3rd grade 963 12.9 
4th grade 979 13.1 
5th grade 828 11.1 
6th grade 996 13.3 
7th grade 922 12.4 
8th grade 748 10.0 
9th grade 301 4.0 
10th grade 267 3.6 
11th grade 165 2.2 
12th grade 105 1.4 


Note. Item percentages may not total 100% because of missing input from some respondents. 
P S ly § inp 


Table 20. CCLC Student Attendance 


Indicate the number of days the student 


has participated in the program: Percentage 
Less than 30 19 
30-59 38.1 
60 - 89 574 
90 or more 74.0 


Note. Item percentages may not total 100% because of missing input from some respondents. 
P S ly § inp 


School-day teachers were also asked how 21st CCLC student behavior had changed in 
six key behavioral areas (Table 21). Most indicated that 21st CCLC student behavior Improved 
or Did Not Need to Improve. Specifically, the 21st CCLC student behavior Jmproved in 
“academic performance” (63%), “motivation to learn” (54%), “turning in his/her homework on 
time” (53%), and “getting along well with other students” (41%). Very few reported a Decline 


in student behavior. 


Virginia 21*' CCLC 2017-2018 Evaluation 58 


Table 21. 218‘ CCLC Student Behavior Changes 


aramnasententiasour student improved | PMLNOENC No change Decnd 
Attending class regularly. 29.9 43.6 23.3 2.7 
Getting along well with other students. 40.8 33.1 22.7 3.1 
Academic performance. 62.6 13.3 20.0 3.8 
Turning in his/her homework on time. 52.5 21.4 22.0 3.7 
Behaving well in class. 39.9 31.6 23.8 43 
Motivation to learn. 53.7 18.8 23.1 3.6 


Note. Item percentages may not total 100% because of missing input from some respondents. 
P S ly § inp 


Largely, school-day teachers indicated overall improvement for students in academics 
and classroom behavior. About three-quarters of teachers agreed that students’ homework 
completion and class participation improved, as well as classroom behavior (Table 22). 


Table 22. 218‘ CCLC Student Overall Improvement 


Overall Improvement Yes No 
Overall, did this student's homework completion and class participation 165 22.6 
improve? : ; 
Overall, did this student's classroom behavior improve? 69.5 28.9 


Note. Item percentages may not total 100% because of missing input from some respondents. 
P rf ly § inp 
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Conclusions 

Overall conclusions are presented below by evaluation question. 

1. What is the nature of the Virginia 21st CCLC grant program and level of participation by 

students? 

Both 21‘ CCLC students and their regular school-day teachers reported positive views of the 
program. Most students are motivated to participate in the afterschool program because it helps 
them do better in school, the activities are fun, and because they enjoy it. Students also feel good 
about the staff and the opportunities being provided for them. While not as many reported that 
they enjoy school more because of the program, most feel they are better readers, better at math, 
and get better grades in school because they attend the 21‘ CCLC afterschool program. 

Regular school-day teachers reported favorable changes in behaviors in many of their 21“ 
CCLC students. The most commonly reported improvements were academic performance, 
motivation to learn, and turning in homework on time. Very few reported a decline in their 21* 
CCLC student’s classroom behavior. 

The level of students who regularly attended center programs (i.e., in attendance for a 
minimum of 30 days) was similar to years past, with the majority in grades three through eight 
and nearly an equal ratio of girls to boys. Most of the students served were African American 
and economically disadvantaged, which directly ties into one of the main purposes of the 21* 
CCLC program — “‘to help students, particularly students who attend high-poverty and low- 
performing schools.” A small portion were identified as having limited English proficiency 


and/or having special needs or disabilities. 
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2. To what degree did centers meet Virginia’s objectives for the program? 
Objective 1: Improve Student Academic Achievement in Reading 

Based on the statistical analyses for grades four through eight that included two years of 
test data, participation in the 21‘' CCLC program was not a statistically significant or 
substantively important (i.e., educationally meaningful) predictor of reading achievement 
outcomes based on either proficiency levels or standardized SOL scaled scores. The number of 
days of 21‘ CCLC participation was also not a statistically significant or substantively important 
predictor of reading outcomes. 

Students in 21‘ CCLC receiving special education services statistically significantly 
outperformed control students on reading proficiency levels, but the magnitude of the effect was 
not substantively important. 

It should be noted that the predictor variables included in the statistical analyses could not 
explain all of the variance (i.e., variability) in reading achievement. In other words, additional 
variables not included in these models (e.g., student motivation, parental involvement) could be 
accounting for some of the variability in reading achievement. 

Results of the descriptive analyses of outcomes for students in grade three who did not 
have prior-year test scores available showed that overall, third-grade 21° CCLC participants in 
2017-2018 were outperformed by non-participants in reading proficiency for all students 
combined and 13 out of 15 available subgroupings and by Virginia in reading proficiency for all 
students combined and 14 out of 15 subgroupings. In terms of SOL scaled scores, third-grade 
21% CCLC participants in 2017-2018 were outperformed by non-participants overall and in 12 
out of 15 subgroup comparisons in reading and by Virginia for all students combined and in 14 


out of 15 subgroupings. 
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Objective 2: Improve Student Academic Achievement in Mathematics 


Based on the statistical analyses for grades four through twelve that included two years of 


test data, participation in the 21‘ CCLC program was not a statistically significant or 
educationally meaningful predictor of mathematics achievement outcomes based on either 
proficiency levels or standardized SOL scaled scores. The number of days of participation was 
negligible, but statistically significant positive predictor of mathematics outcomes. 


Students in 21“ CCLC receiving special education services statistically significantly 


a 


outperformed control students on mathematics proficiency levels, but the magnitude of the effect 


was not substantively important. Economically disadvantaged students in 21*' CCLC statistically 


significantly outperformed control students on mathematics standardized SOL scaled scores, 
while non-economically disadvantaged control students outperformed 21*' CCLC students, but 
the magnitude of the effects were not substantively important. 

It should be noted that the predictor variables included in the statistical analyses did not 
explain all of the variance in mathematics achievement. Therefore, additional variables not 
included in these models could be accounting for some of the variability in mathematics 
achievement. 

Results of the descriptive analyses of outcomes for students in grade three who did not 
have prior-year test scores available showed that in terms of mathematics proficiency, third- 
grade 21 CCLC participants outperformed non-participants on 10 out of 15 subgroupings. 
However, the Commonwealth outperformed 21*' CCLC students for all students combined and 
on 12 out of 15 subgroupings. In terms of SOL scaled scores, third-grade non-participants did 


better overall and in 7 out of 15 subgroups in mathematics compared to 21*' CCLC participants 
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while the Commonwealth did better overall and in 13 out of 15 subgroupings compared to 21* 
CCLC students in 2017-2018. 

When interpreting the findings on the impact of 21‘ CCLC participation on student 
academic achievement in reading and mathematics, readers should keep in mind that while 21“ 
CCLC participants (i.e., students having 30 days or more of attendance) were compared to 
controls (i.e., students who were eligible to participate, but had zero days of attendance in the 
21st CCLC program), there is no information available on what, if any, services or programs 
other than 21*' CCLC control students may have participated in. That is to say, it is possible that 
21 CCLC participants were compared to students who, while not attending a 21 CCLC 
program, did participate in some other form of enrichment or tutoring, meaning that the 
comparison was actually between multiple forms of intervention (i.e., 21‘* CCLC vs. other types 
of student support) as opposed to 21‘ CCLC participation vs. nothing. 

Another issue to consider in assessing the impact of 21‘ CCLC participation on academic 
achievement (or regular school-day attendance) is that students attending the average number of 
days (n = 35.5 across all grades 3 — 12), if they received one hour of tutoring per day, would 
have only received 3.6% more hours (or 6 2 more days) of instruction over the standard school 
year (of 990 instructional hours or 180 instructional days respectively). It is unclear what 
influence that limited amount of additional instruction could have on student outcomes. 

Improve Student School-Day Attendance. For students in grades four through twelve, 
the school-day attendance analyses showed a statistically significant positive impact of 21“ 
CCLC participation (“Yes” or “No’”) on school-day attendance for the overall sample, although 


the effect was not substantively important. In addition, students in 21*'* CCLC had statistically 


? https://law.lis. virginia.gov/admincode/title8/agency20/chapter 13 1/section150/ 
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significantly higher school-day attendance compared to control students in the subgroups (a) not 
limited English proficient, (b) special education, (c) not special education, and (d) economically 
disadvantaged. However, the magnitude of the effects was not substantively important. 
Objective 3: Provide Opportunities for Parent Education 

Objective 3 directly ties into one of the purposes of the 21‘' CCLC program — “To offer 
families of students served by community learning centers opportunities for literacy and related 
educational development.” As a whole, 4,800 parents and adult family members were served 
during the 2017-2018 school year, including 537 parents and adult family members participating 
over the summer. Moreover, subobjectives were almost always met (>87%) except for the 
General Educational Development (GED) certificate program, which was only met by half of 
those programs that chose it as a subobjective. 

3. In what ways do attendance at a 21st CCLC, type of and time allocated to activities, and 
hours of operation predict academic achievement? 

The results suggest that a higher number of center staff had a statistically significant and 
positive, yet small impact on both reading proficiency level and standardized SOL scaled scores. 
The number of days attended was a statistically significant positive predictor of both 
mathematics proficiency level and mathematics standardized SOL scaled score outcomes, 
although the magnitude of the effect was negligible. No other center-level variables had a 
statistically significant impact on students’ academic achievement. 

It should be noted that the predictor variables available and included in the statistical 
analyses did not explain all of the variance (1.e., variability) in either reading or mathematics 
achievement outcomes. Consequently, additional variables not included or available in these 


models could be accounting for some of the variability in achievement. 
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4. What promising practices regarding the achievement of required objectives were 
identified by centers? 

Among comments submitted by grantees across the six subjects (math and 
reading/English; parent education; character education; enrichment opportunities; and 
community partnerships), the most heavily emphasized “promising practices” addressed three 
broad areas. First and most prominently, there were practices that supported the students. These 
can be broken into three types: Support for academic performance, enrichment activities, and 
character education. The second broad group of practices encompassed “parent support,” which 
fell into four types: Helping parents assist their children, enhancing parent employability, 
enhancing other life skills, and practices intended to support parents’ continuing attendance and 
participation in the program. Finally, there were practices aimed at improving community 


partnerships. 
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Appendix A: Supplemental Program Objectives 

In addition to the state mandated 21‘' CCLC program objectives, some grantees chose 
supplemental objectives as part of their center activities. This appendix provides information on 
the percentage of centers choosing each supplemental objective and the success centers reported 
in meeting these objectives. 
Provide Enrichment Opportunities 
Providing enrichment opportunities was selected by some 21‘ CCLC grantees as a supplemental 
objective. Nearly all (88%) wanted to increase children's depth of understanding of academic 
subjects by non-traditional instruction through this subobjective. Of those who selected to 
provide enrichment opportunities, 93% reported meeting this supplemental objective. The 
percentage of centers selecting subobjectives within this objective are shown in Table A-1. 
Success of the reporting centers in meeting these subobjectives is presented in Table A-2. 


Table A-1. Centers Selecting Subobjectives for Provide Enrichment Opportunities 


Provide Enrichment Opportunities Subobjective Percentage 


Increase children's exposure to the fine arts and cultural events. 71.4 


Increase children's depth of understanding of academic subjects through non- 


traditional instruction, such as STEM activities, and problem-based learning. Slee 
Increase children's health awareness and physical education. 77.7 
Provide college and career readiness programs. 35.7 
Provide community and service learning programs. 42.0 
Other 3.6 
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Table A-2. Success in Meeting Subobjectives for Providing Enrichment Opportunities 


Enrichment Opportunities Subobjective Average Number 
How many opportunities were provided for students to participate in fine arts and 34.2 
cultural events? : 
How many opportunities were provided for students to participate in non-traditional 54.4 
instructional activities? ; 
How many opportunities were provided for students to participate in health 15.8 
awareness and physical education activities? ; 
How many opportunities were provided for students to participate in college and 33.9 
career readiness programs? ; 
How many “Other” enrichment activities did students have an opportunity to 41.0 
participate in? ; 


Improve Community Partnerships 

Improving community partnerships was also selected by some 21“ CCLC grantees as a 
supplemental objective. About three-fourths (73%) wanted to increase the activities of 
community partners through this subobjective. Of those who selected improving community 
partnerships, all reported meeting this supplemental objective. The percentage of centers 
selecting subobjectives within this objective are shown in Table A-3. Success of the reporting 
centers in meeting these subobjectives is presented in Table A-4. 


Table A-3. Centers Selecting Subobjectives for Improving Community Partnerships 


Improving Community Partnerships Subobjective Percentage 
Increase the number of community partners 54.2 
Increase the activities of community partners 72.9 
Improve communication with community partners 67.8 
Improve the sustainability of the program through community partner commitments 32.2 
beyond the grant period j 
Other 17 
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Table A-4. Success in Meeting Subobjectives for Improving Community Partnerships 


eda Percent Percent 

Subobjective Met Not Met 
Did the number of community partners increase? 87.5 12.5 
Did the activities of community partners increase? 95.3 4.7 
Did communication with community partners increase? 95.0 5.0 


Has the potential sustainability of the program increased through 


community partner commitments? aa a 
Did the "Other" subobjective help improve community partnerships? 100.0 0.0 
Overall, did your site's 21st CCLC program meet its objective to 91.5 85 


improve community partnerships? 


Provide Character Education 

Providing character education was another supplemental objective selected by some 21" 
CCLC grantees. Nearly all (87%) wanted to improve classroom behavior through this 
subobjective. Of those who selected to provide character education, 93% reported meeting this 
supplemental objective. The percentage of centers selecting subobjectives within this objective 
is shown in Table A-5. Success of the reporting centers in meeting these subobjectives is 
presented in Table A-6. 


Table A-5. Centers Selecting Subobjectives for Provide Character Education 


Provide Character Education Subobjectives Percentage 

Improve classroom behavior 86.6 
Submits completed homework on time 62.7 
Improve classroom participation 55.2 
Improve class attendance 61.2 
Improve motivation to learn 62.7 
Improve ability to get along with other students 67.2 
Provide programs in character education, such as preventing drug/alcohol use and/or 44.8 
violence, truancy, and youth leadership ; 

Other 1.5 
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Table A-6. Success in Meeting Subobjectives for Provide Character Education 


Provide Character Education Subobjectives dors Percent 
Not Met 
Was classroom behavior improved among students attending 30 days 983 7 
or more? ‘ ; 
Did submission of completed homework on time improve for students 
. 100.0 0.0 
attending 30 days or more? 
Did classroom participation behavior improve for students attending 
94.6 0.0 
30 days or more? 
Did classroom attendance improve for students attending 30 days or 878 22 
more? ; ; 
Did motivation to learn improve for students attending 30 days or 976 24 
more? : : 
Did ability to get along with others improve for students attending 30 911 67 
days of more? ; ; 
Did the programs in character education, such as preventing 
drug/alcohol use and/or violence, truancy, and youth leadership have 90.0 6.7 
a positive impact on students attending 30 days or more? 
Did the "Other" subobjective improve for students attending 30 days 100.0 0.0 
or more? j ; 
Overall, did your site's 21st CCLC program meet its objective to 
; : 92.5 6.0 
provide character education? 


Virginia 21° CCLC 2017-2018 Evaluation (Appendix A) 


70 


